Generalized entropies through Bayesian estimation

The demand made upon computational analysis of observed symbolic sequences has been increasing in the last decade. Here, the concept of entropy receives applications, and the generalizations according to Tsallis H (T) q and R enyi H (R) q provide whole-spectra of entropies characterized by an order q. An enduring practical problem lies in the estimation of these entropies from observed data. The nite size of data sets can lead to serious systematic and statistical estimation errors. We focus on the problem of estimating generalized entropies from limited data samples and derive a Bayesian estimator of the Tsallis entropy, H (T) q , including the (q = 1) Shannon entropy. By extending our previous results on statistical entropy estimation of symbol sequences 12], we use a prior distribution over the probabilities which is of Dirichlet-type. Using the relationship between H (T) q and H (R) q , we utilize the Bayesian entropy estimator H (T) q to estimate the R enyi entropy H (R) q from observed data. The Bayesian estimator yields the smallest mean-squared deviation from the true parameter as compared with any other estimator. We compare the Bayesian entropy estimators with the frequency-count estimators of H (T) q and H (R) q. Numerical simulations reveal that the Bayesian entropy estimator reduces statistical estimation errors of generalized entropies for statistical processes such as generated by higher-order Markov models.